File API Design Decisions

In this lesson, we examine the overall operation of the file API and the interaction of the individual components. This will help us make certain design decisions while diving deeper into the design challenges.

Design overview#

There are many architectures that lead to good API design, but we can’t simply pick one and start developing. API design is often influenced by business requirements, technologies used, etc. Therefore, defining a simple step-by-step guide for choosing an architecture may not be possible. Often, a combination of multiple architectures work in tandem to achieve organizational goals. Let’s go over the design considerations of a file API and determine the suitable ingredients (architecture style, data format, etc.) to satisfy our needs.

The following diagram contains the detailed workflow of the components and services involved in the file API design.

Let’s look at the functionality of these components and services in the table below.

Workflow#

For a better understanding, we can divide the workflow shown in the illustration above into the following two parts.

• Client to API gateway: Before performing any operation (upload, download, and so on), the API needs to authenticate the client and ensure that it has read/write access to the data. After successful verification, a user can upload, download, list, or delete a file from the server through HTTP requests.

• API gateway to downstream services: As the request reaches the API gateway, the gateway forwards the request to the appropriate API services. If the request is to upload a file, the application server extracts the metadata (such as user ID, file name, file size, file location, etc.) from the incoming request and sends it to the user to file mapping system (UFMS), which adds it in a relational (SQL) database. The actual content of the file is stored in temporary storage. This data is further processed (compressed, encrypted, and so on) and stored in the blob storage using a storage-optimal format.
  Likewise, to download a file, the client requests the API gateway, which forwards its request to the appropriate application server that asks the UFMS for relevant information. After the server receives the required information, it retrieves the specified file contents, compiles the response, and returns it to the user. The server can also refer to the file contents present on the regionally distributed CDN servers for fast delivery.

Note: After data is successfully stored in blob storage, multiple copies of it are made and sent to different replica storage spaces (backup servers and CDN servers) for faster delivery, reliability, and disaster management.

Design considerations#

Now that we have a solid understanding of how the file API service works, let's go a step further and decide on the architectural style, data format, and HTTP version that we will use in our API.

Architectural style#

Let's consider the architectural styles we discussed in this course and see which works best for our file service:

• REST: This provides a well-defined, standardized way to manage resources. Our API performs the most basic operations on a single resource (file), and REST can be a potential candidate because it provides this functionality well.

• gRPC: This is best suited for communication-intensive systems because it supports the management of multiple request-response communication links (subchannels) over a single HTTP/2.0 connection (channel). It requires persistent connections and maintains connection states at each end to reduce the TCP handshake overhead. These features may not be fully utilized in our file API because individual client uploads may not be as intensive, and clients may not send multiple requests on a consistent network. Furthermore, gRPC is an action-oriented architectural style, which may add additional complexity to our design.

• GraphQL: This excels at managing different resources with a single request from multiple endpoints. It also efficiently retrieves the required data to address overfetching/underfetching in RESTful APIs. However, interacting and retrieving data from multiple resources is not a use case for our service.

The discussion above suggests we should use REST because it has better support for different operations in our file API and avoids unnecessary complexity. Therefore, we propose REST as a communication protocol from client to API gateway and from API gateway to downstream services.

Data formats#

Let's consider the data formats we discussed in this course and see which suits our file service requirements the most:

• JSON: This is human readable, compact, and easy to implement, with a built-in JavaScript (the most widely used scripting language for web services) support parser. We need to send user information, file information, and file data between the client (web application) and the server. For file and user-related metadata, we need readability for third-party developers to easily integrate our API into their applications. JSON meets all these requirements, making it a suitable format for our API.

• XML: This also supports object representations and can represent more complex data than JSON, but it has some major drawbacks, such as larger encoded file sizes and a lack of built-in support for parsers, which makes it less suitable for our API service.

• Binary: These formats are fast to process and compact in size compared to textual representations, but these formats sacrifice human readability, making it difficult to debug problems and understand errors. Our API needs human interactivity for client-server interaction and debugging issues. However, we can use binary formats to transfer, store, and retrieve actual file content.

As mentioned above, JSON provides everything we need and is also suitable for the RESTful API style. So, JSON is the data exchange format from client to API gateway and from API gateway to downstream services.

Remember: The actual file content is transmitted in binary format. Only the requests and responses are formatted in JSON for processing.

HTTP version#

We are using HTTP upload requests to transfer files between the client and the web server (more on this in the next lesson). Let's discuss which HTTP version we should use as the communication protocol in the file service and why.

• HTTP/1.1: This supports range/chunked data transfers, using TCP and TLS for reliable and secure transferring. It also has the advantage of being supported by existing infrastructure. Our main focus is fast, secure, and reliable data transfers between client and server machines. HTTP/1.1 fits our requirements very well, making it a suitable option for a file API.

• HTTP/2.0: This supports several additional features, primarily multiplexing streams for faster bidirectional communication over a single connection. However, sending a single file does not show significant improvement in performance as compared to HTTP/1.1, as shown by the following illustration.
  From the illustration below, we can see that there is no significant difference in the upload process of a single file. HTTP/2.0 seems to excel when it comes to uploading multiple files simultaneously. It also overcomes the head of line (HOL) blocking issue of HTTP/1.1. HTTP/2.0 can be considered for future work when our service adds the multifile upload feature.

• HTTP/3.0: This was released very recently and is gaining quite a bit of traction in the industry. However, HTTP/3.0 has low infrastructure support, which increases development costs and narrows the range of supported clients. So, using HTTP/3.0 for slight improvements with added costs may not be a good choice.

Considering the information above, HTTP/1.1 is a suitable option that can meet all our functional needs. Also, upgrading from HTTP/1.1 to HTTP/2.0 only requires clients and servers to support HTTP/2.0 connections, while the rest of the application logic can remain the same. We are adding support for both HTTP/1.1 and HTTP/2.0 to our file API. However, our discussion will mainly focus on HTTP/1.1, which is the minimum requirement for our API to function. We will also explore possible cases where HTTP/2.0 outperforms HTTP/1.1.

Summary#

A summary of our design decisions is given in the table below: